Information retrieval is the area of study concerned with searching for documents, information within documents, and/or metadata about documents. Accordingly, a search engine can be referred to as an information retrieval application. A web search engine is designed to search for information on the World Wide Web and FTP servers. Search engines include rankers that are configured to receive a query and output a ranked list of search results responsive to receipt of the query. The search results may include web pages, images, information and other types of data. Additionally, a search engine may be configured to mine data that is available in network-accessible databases and/or open directories.
Generally, a ranker is trained by optimizing performance of the ranker in accordance with a performance metric, such that parameters of the ranker are determined to cause the ranker to perform as well as possible with respect to some performance metric (on average across all queries in training data). Labeled training data may be retained in a data store, such that the training data includes queries and (labeled) ranked search results that correspond to such queries. Therefore, if the ranker is operating ideally, for a given query the ranker will output search results in the order identified in the training data. Various techniques are currently available to compute the performance of a ranker for training data. Exemplary techniques for determining performance of the ranker on a per query basis include normalized discounted cumulative gain (NDCG), mean average precision (MAP), amongst others.
Accordingly, the general performance of a ranker is often determined by averaging the per query performance score. Thus, conventionally, parameter values of the ranker are learned such that the averaged per query performance score is as high as possible. Such approach, however, fails to take into consideration the perception of a user when the search engine performs very poorly for particular queries. For instance, even if a ranker is learned as described above (where parameter values are set to cause the average performance scores across queries to be maximized), if the ranker performs very poorly on certain queries, the user will perceive that the ranker generally performs poorly.